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About the research 


Does 1 = 1? Mapping measures of adult literacy and numeracy 

Michelle Circelli, NCVER; Shelley Gillis, Work-based Education Research Centre, 
Victoria University; Mark Dulhunty, Margaret Wu and Leanne Calvitto, Educational 
Measurement Solutions 

Being able to measure the level of proficiency in literacy and numeracy skills, and any changes in the 
level of skills, is important for getting a sense of how well language, literacy and numeracy programs 
are working. Among the tools used to measure language, literacy and numeracy proficiency in Australia 
are the Adult Literacy and Life Skills (ALLS) survey and the Australian Core Skills Framework (ACSF). 

The Adult Literacy and Life Skills survey measures the skills of adult populations within and across a 
number of participating Organisation for Economic Co-operation and Development (OECD) countries. It 
is used by the Australian Government to monitor progress against the National Skills and Workforce 
Development Agreement. Furthermore, the next iteration of this survey will be used to measure the 
success of the 2012-22 National Foundation Skills Strategy for Adults. However, the survey has two 
drawbacks: it is a relatively coarse measure and is designed to provide a summary of literacy and 
numeracy rather than to act as an assessment tool; and it is only administered every ten years. 

Contrasting with the Adult Literacy and Life Skills survey, the Australian Core Skills Framework - used 
in two key federal government adult language, literacy and numeracy programs, as well as in a variety 
of other settings, including the South Australian Certificate of Education - can be applied at the 
individual level and provides evidence of progress, such that a learner’s performance in a core skill 
can be assessed, and strengths and weaknesses identified. Further, data about a learner’s 
performance can be gathered at frequent intervals. 

Both these frameworks have five performance levels and it is sometimes assumed that these levels 
are equivalent. But are they? This paper presents findings from a study that looked at the issue of the 
equivalence of the frameworks. 

Key messages 

■ Equivalence between the two frameworks at the lowest skill level was found - one does equal 
one. However, the alignment was not as direct at the higher skills levels, with the numeracy and 
reading constructs of the Adult Literacy and Life Skills survey found to be generally more complex 
than those of the Australian Core Skills Framework. Indeed, Level 3 ALLS - the minimum 
aspirational target of the National Foundation Skills Strategy for Adults - was similar in complexity 
to exit Level 4 of the ACSF. 

■ A definite hierarchical structure within the levels of the Australian Core Skills Framework was 
confirmed, offering the potential to clearly demonstrate progress within a level. 

This research has shown that alignment between the two frameworks is achievable and that this 
alignment offers the potential for measuring progress against national objectives more regularly. 
Indeed, the ACSF offers a way of monitoring any improvements in adult literacy and numeracy in a 
more nuanced manner. 


Tom Karmel 

Managing Director, NCVER 
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Does 1 = 1? Mapping measures of adult literacy and numeracy 


Introduction 


Around half of Australia’s adult population have low literacy and numeracy skills, as measured by the 
Adult Literacy and Life Skills survey (ALLS; ABS 2008). ‘Low’ skills refers to those who fall into Levels 
1 and 2 of the five performance levels of this survey. Level 3 is considered to be the minimum needed 
by individuals in order to meet the complex demands of everyday life and work in a knowledge-based 
economy (Statistics Canada 2005). This proportion is largely unchanged from the previous 
international literacy survey of the mid-1990s (ABS 1996). Results from the next iteration of the 
international adult literacy and numeracy survey - the Programme for the International Assessment of 
Adult Competencies (PIAAC) - are due for release in late 2013 and it is with interest that we await 
these to see what, if any, changes have occurred. 

The magnitude of the low literacy skills problem among adults in Australia is similar to that in 
comparable, mainly English-speaking, countries, including New Zealand, Canada, the United States 
and the United Kingdom. 

We know that those with low literacy and numeracy skills are more likely to: 

■ have lower educational attainment (ten years or fewer of formal education) 

■ be unemployed or not looking for work (that is, out of the labour force) 

■ be older (45 years and older) 

■ be from non-English speaking backgrounds. 

In addition to the Adult Literacy and Life Skills survey, there are a number of other tools used to 
measure language, literacy and numeracy proficiency. Among these is the Australian Core Skills 
Framework (ACSF). This is routinely used in key federal government programs such as the Workplace 
English Language and Literacy (WELL) program and the Language, Literacy and Numeracy Program 
(LLNP) to assess the state and progress of individual or group literacy and numeracy skills. Both the 
Australian Core Skills Framework and the Adult Literacy and Life Skills survey have five levels of 
performance; it is sometimes assumed that these levels are equal. But are they? 

Why was this project undertaken? 

In late 2008, as part of the National Skills and Workforce Development Agreement, a Council of 
Australian Governments (COAG) directive specified that the proportion of the working-age population 
with low foundation skill levels be reduced to enable effective educational, labour market and social 
participation, and that the proportions at ALLS Levels 1, 2 and 3 be monitored as a means of checking 
progress. That is, the objectives were stated in terms of ALLS survey levels. 

Further, during the course of this research the National Foundation Skills Strategy for Adults 1 was 
released - the first such strategy in 20 years - which is focused on improving outcomes for working- 
age Australians. The performance measure for this strategy will be ‘by 2022, two thirds of working 
age Australians will have literacy and numeracy skills at Level 3 or above’ (Standing Committee on 
Tertiary Education, Skills and Employment 2012, p.10). Level 3 here is in reference to the levels in 


1 See <http://www.scotese.natese.gov.au/ data/assets/pdf_file/0007/71755/National_Foundation_Skills_ 

Strategy_for_Adults.pdf >. 
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the Adult Literacy and Life Skills survey and the Programme for the International Assessment of 
Adult Competencies. 

As noted above, two key federal government programs which use the ACSF are the Workplace English 
Language and Literacy and the Language, Literacy and Numeracy Programs. These programs provide 
information on a very small proportion of the population (approximately 100 000 per year) who fall 
within the COAG target area. There are many state-level programs that could also be used to provide 
further information on the literacy and numeracy progress of various learner groups. While these 
programs cannot be measured and reported using the ALLS or PIAAC tests, they could be monitored 
against the ACSF benchmarks. 

The aim of this project is to investigate whether the reading and numeracy performance levels of the 
Adult Literacy and Life Skills survey and the Australian Core Skills Framework can be aligned, 2 
essentially to determine whether or not the ACSF performance levels could be used as a proxy for 
ALLS performance levels. 3 This would make it possible to provide information on the literacy and 
numeracy development of identified target groups of the adult population on a more frequent basis 
than is currently available from the large-scale international testing programs. 

Measuring adult literacy and numeracy using the ALLS and ACSF 

The Adult Literacy and Life Skills survey and its predecessor, the International Adult Literacy Survey 
(IALS), were developed to enable the collection of comparable international data on literacy and 
numeracy proficiency. Twenty years ago, the Organisation for Economic Co-operation and Development 
(OECD) recognised that low literacy levels were having a significant impact on economic performance 
and social cohesion at an international level. But a lack of data at that time meant attempts to gain a 
better sense of the extent of literacy problems, and the policy implications that would arise from this, 
were unsuccessful (cited in National Center for Educational Statistics 1998, p. 1 3). 

The focus of the International Adult Literacy Survey, the Adult Literacy and Life Skills survey, and the 
current survey, the Programme for the International Assessment of Adult Competencies, is always on 
the skills an individual needs to participate fully and successfully in a modern society. Such surveys 
are designed to provide performance information at aggregate levels such as the adult population and 
by important sub-groups (for example, gender, location). Given the cost associated with the 
management and administration of such large-scale international surveys, there is generally a longer 
period of time between surveys (five to ten years). In Australia, the IALS was administered in 1996, 
the ALLS survey in 2006, and PIAAC was undertaken in late 201 1 -early 2012. While these types of 
surveys provide important information about Australia’s skills position relative to other countries, this 
timeframe does not necessarily permit the close monitoring of progress against national goals. 

The Australian Core Skills Framework describes performance in the five core skills of reading, writing, 
oral communication, numeracy and learning. 4 It is intended to act as a national framework for 
describing and discussing English language, literacy and numeracy performance and contains 
benchmarks against which to assess and report on the progress of individuals or learner cohorts. 


2 A mapping exercise of the National Reporting System and the IALS was undertaken in 2002, as it seemed, on the 
surface at least, that the two measures were directly comparable, as both had five levels of competence. A general 
alignment was found but not a direct one-to-one relationship between the five levels of each scale. Indeed, there were 
no IALS tasks at NRS level 1 (Hagston 2002). 

3 It is important to note that the purpose of this project was not to evaluate the utility of either the ACSF or ALLS but 
rather to evaluate their possible commensurability. 

4 See <http://www.innovation.gov.au/Skills/LiteracyAndNumeracy/AustralianCoreSkillsFramework/Pages/default.aspx>. 
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During the course of this project a revised version of the ACSF was released which includes a Pre- 
Level 1 Supplement. 5 This supplement allows for the identification of core skill requirements for 
individuals with very low-level literacy and numeracy skills. 

In addition to being used in the Language, Literacy and Numeracy and the Workplace English Language 
and Literacy programs, the ACSF is now being adopted across a range of contexts and for a range of 
purposes. For example, the South Australian Certificate of Education (SACE) Board endorsed the ACSF 
Level 3 descriptions in reading and writing as reference points for the SACE literacy benchmark, while 
Victoria University has adopted the ACSF as part of its whole-of-university strategy to support 
students’ literacy and numeracy skills development. In the vocational education and training (VET) 
sector, several industry skills councils are sponsoring national professional development around the 
ACSF for trainers in their fields and have mapped or are currently mapping training package units to 
the framework. 

A key difference between the Adult Literacy and Life Skills survey and Australian Core Skills 
Framework is the assessment purpose. The large-scale ALLS survey is a summative and evaluative 
tool. That is, it is used to give a summary of the knowledge and skill levels of a population, or sub- 
population, at a point in time and does not provide feedback to inform future learning. 

The ACSF can be used as either a summative or a formative tool. At any point in time, a learner’s 
performance in a core skill can be measured against the descriptors (called ‘Indicators’ and 
‘Performance Features’) associated with each of the five levels, and a level of performance assigned. 
The ACSF can also be used as a formative or diagnostic tool. Any activity or test can become an 
assessment instrument if it is mapped to the ACSF and then used to identify an individual’s specific 
strengths and weaknesses. The Performance Features offer a means of providing detailed 
performance feedback and of identifying where the focus of subsequent effort might yield useful 
results. Progress over time can be monitored against the levels and also against specific Indicators and 
Performance Features. 

There is increasing interest in the summative capacity of the Australian Core Skills Framework. For 
example, the Australian Council for Educational Research (ACER) has aligned its Core Skills Profile for 
adults to the ACSF, while, as noted above, Victoria University uses activities based on the ASCF to 
establish language, literacy and numeracy performance benchmarks for commencing students as a 
precursor to tracking, monitoring and measuring performance improvement over time. 


5 While this project used the revised version of the ACSF (published 2012) to undertake the alignment exercise, the 
alignment of the Pre-Level 1 Supplement to the ALLS was outside the scope of the project. 
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Method 


There were two stages to this project. The first was a study undertaken in 2010 (Circelli, Curtis & 
Perkins 2011) to determine whether a potential mapping between the two frameworks was feasible. 
This involved an expert group, which included developers of the Australian Core Skills Framework, 
along with an experienced item developer and a literacy practitioner, assessing the position of a 
number of prose and document literacy and numeracy ALLS items in the ACSF structure, 6 based on the 
assumption that a learner would attempt to perform the tasks independently. Since ALLS items have 
known locations on the relevant ALLS scale, the consensus judgment of panel members provided a 
qualitative link between the two scales. 7 

For this phase, items that represented Levels 1 and 2 and the lower part of Level 3 of the ALLS prose 
and document literacy and numeracy scales were used, since individuals whose literacy performance 
lies within this range have tended to be of most interest in programs that use the ACSF as a tool for 
literacy improvement. 8 At the completion of the study, there was general consensus among the 
participants that the mapping process was feasible for the reading domain of the ACSF to the ALLS 
prose and document literacy domains - hereafter collectively referred to as the reading construct - 
as well as the numeracy domains of the two frameworks. 

The second stage of the project, the focus of this report, involved a larger-scale research study to 
empirically align the two frameworks to a single scale for reading and numeracy using Item Response 
Theory (Rasch I960). 9 

During a 15-minute online survey, teachers/tutors familiar with adult literacy and numeracy concepts 
anonymously rated a learner whose literacy and/or numeracy levels were most familiar to them 
against statements and sample tasks - collectively referred to as ‘items’ - drawn directly from both 
the ACSF and ALLS frameworks. The ALLS items comprised each of the reading and numeracy Level 
Descriptors, 10 as well as a sample of publicly available retired scaled items and a random sample of 


6 For the feasibility phase a version of the Delphi method (Linstone & Turoff 2002) was used. Further information about 
the Delphi method is provided in Circelli, Curtis and Perkins (2011), together with qualitative descriptions of 
performance at the various levels of the two frameworks and a brief description of the ALLS items used in this study. 

7 In the first phase of the research the focus was on establishing whether qualitative mapping is feasible before possibly 
committing to the complexity and cost of a quantitative mapping activity. 

8 The items used were developed by Statistics Canada and the Educational Testing Service (ETS). Items used in IALS, 
ALLS and PIAAC are confidential as it is important to be able to use common items across assessment occasions in 
order to monitor progress in literacy achievement over time. Statistics Canada and the ETS allowed NCVER to use the 
items, subject to confidentiality undertakings being entered into by all project participants. The items used in this 
study were selected by Statistics Canada and the Educational Testing Service and are not required for the PIAAC. 
NCVER is grateful to Statistics Canada and the ETS for permission to use their items and for their assistance in 
selecting items for this project. 

9 Item Response Theory (IRT) encompasses the design, analysis and scoring of tests, questionnaires and similar 
instruments for measuring abilities, attitudes, or scales that describe a particular concept, in this case, adult 
literacy and numeracy. It is based on the application of related mathematical models to testing data. IRT 
underpinned the development of the ALLS scale and is also the preferred method for the development of tests such 
as the National Assessment Project - Literacy and Numeracy (NAPLAN) and the OECD Programme for International 
Student Assessment (PISA). IRT was considered to be the most direct method for determining and comparing the 
complexity of the two frameworks. 

10 The Level Descriptors comprised publicly available statements that defined each level for reading and numeracy. 

The level descriptors included both the original statements published by Statistics Canada along with statements 
modified by the expert panel of this current project. Refer to ‘Panelling the item content’ in the Survey Design 
section of the Mapping adult literacy performance: support document. 
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Numeracy Complexity Statements. 11 In relation to the ACSF, a random sample of Performance 
Features and the total pool of level Indicators were selected. 12 The survey item pool comprised a total 
of 79 items for reading (34 items representing the ALLS and 45 representing the ACSF) and 86 items 
for numeracy (50 items representing the ACSF and 36 representing the ALLS). 

There were six forms of the survey (three forms each for reading and numeracy) with link items (that 
is, common items across forms to enable the forms to be equated onto a single scale) to minimise 
respondent workload and at the same time enable the collection of sufficient data on all 79 items for 
reading and 86 items for numeracy, based on the expected sample size of respondents. Each form 
contained approximately 50 items, with content drawn from both frameworks across three adjacent 
levels on each framework. Items were presented in random order so the respondents were not able to 
obtain external cues about the level of an item (other than the wording of the item itself), and also to 
avoid any item positioning effect. 13 This meant that the complexity of an item could be determined 
solely by the language contained within each item, as opposed to making an a priori assumption about 
the relative complexity of the item content according to its original positioning within the framework. 

At the start of the survey, each respondent was asked to provide background information as well as to 
select an anonymous learner to form the basis of the ratings. The respondent was first asked to make 
a holistic judgment of the learner’s ACSF level in either the reading and/or numeracy skill area. This 
judgment was used to assign an appropriate form for the respondent to complete. Each respondent 
was then required to use a three-point rating scale (‘not very likely’, ‘somewhat likely’ or ‘very 
likely’) to rate the likelihood that the learner would be able to independently perform the task 
described by each item. 

Data for the numeracy and reading constructs were analysed separately and placed onto the same 
scale of measurement using Item Response Theory (Rasch 1960), which enabled the relative 
complexities of the five levels on the Australian Core Skills Framework and the five levels of the Adult 
Literacy and Life Skills survey to be determined. As the survey design contained items common to 
multiple survey forms, all items across all three forms per concept could be mapped onto the same 
scale using common item equating. The outcome of the analyses was that the complexity of each item 
and the ability of each learner were estimated on a scale measured in logit units. A transformation 
was then applied to these estimates to make them more interpretable for end-users of this report 
(analogous to converting inches into centimetres) with items assigned a complexity estimate ranging 
from 100 (low complexity) to 200 (high complexity). 

To increase the response rate, participants who completed the survey were given the opportunity to 
voluntarily enter into a draw to win one of six iPads (3rd generation). Participants were also 
encouraged to complete more than one survey and, as such, had the opportunity to enter the draw 
each time they completed a survey. 

A more detailed description of the methodology can be found in the support document, Mapping adult 
literacy performance: support document. 


11 Numeracy Complexity Statements underpin the design and development of numeracy items in the ALLS survey and 
express the complexity of the mathematical information/data needed to be manipulated as well as the mathematical 
action required. 

12 ACSF ‘Indicators’ refer to statements that provide an overview of performance at each level of each core skill. ACSF 
‘Performance Features’ refer to statements that provide detailed descriptors of what a person operating at a 
particular level is able to do. 

13 The Item Positioning effect refers to the tendency for items positioned toward the end of a survey to have more 
missing data or guessed responses due to respondent fatigue and/or boredom. 
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Results 


Four hundred and eleven surveys were completed, with most respondents rating against the reading 
concept (72%). Although all states and territories were represented, the majority of respondents were 
located in Victoria and New South Wales, with very few from the Australian Capital Territory (see 
figure A1 in the appendix). Respondents tended to be employed in an education and/or training 
institute (see figure A2 and table A1 in the appendix) and had more than five years experience in 
delivering adult language, literacy or numeracy training. Only 7% of the respondents had less than one 
year’s experience, indicating that the majority were very experienced in this field (see figure A3 in 
the appendix). 

When rating the reading concept, respondents tended to select learners who were taking (or had 
previously undertaken) an English as a Second Language (ESL) program (26%) or were participating in a 
Language, Literacy or Numeracy Program (22%), whereas with the numeracy concept, respondents 
tended to select learners from an adult literacy and numeracy course (32%) or a Language, Literacy or 
Numeracy Program (22%). In both instances, respondents tended to select learners who were at Levels 
1 to 3 on the ACSF (as determined by the their initial holistic judgment), with fewer than 10% of 
respondents selecting learners thought to be at Levels 4 and 5 in either reading or numeracy. 

Subsequently, most ratings were made against items that were aligned to Levels 2 and 3 for both 
reading and numeracy (ranging from 102 to 294 ratings made), with very few ratings made against the 
items that were at Level 5 on both frameworks: 15 ratings per item for numeracy, 39 ratings per item 
for reading. Given the large standard errors of measurement found for the Level 5 items, caution 
should be exercised when comparing the highest complexity levels on both frameworks, particularly 
for the numeracy domain. 

Estimates of learners’ ability were highly reliable in both reading and numeracy, with reliability 
estimates of >0.971 produced for both constructs. 

Are there five distinct performance levels within the ALLS and 
ACSF frameworks? 

Before looking at how the performance levels of the reading and numeracy domains of the Adult 
Literacy and Life Skills survey related to those of the Australian Core Skills Framework, this study 
examined the validity of the items used within each framework. That is, the study looked at the 
extent to which items that were considered to require a lower skill level or were less complex in 
either the reading or numeracy constructs actually were, or the extent to which items considered to 
be more complex - at a higher level of skill — actually were (for example, did items considered to 
measure ACSF Level 1 in the reading construct have lower complexity estimates than those thought to 
be at Level 2? Or were items considered to measure ACSF Level 2 items less complex than those at 
Level 3, and so on). By undertaking this matching exercise between the expected sequencing of the 
items and the empirically calibrated complexity measures within each level for each framework, 
evidence of the validity of the measures used in this study could be garnered. 
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The study found a general pattern of association between an item’s complexity estimate and its 
expected level (as determined by the level it was originally assigned within each framework). 14 That 
is, in general, items thought to be at higher levels were associated with higher empirically calibrated 
complexity, while those items thought to be at lower skill levels were associated with lower 
calibrated complexity. This indicates that both the ALLS and ACSF frameworks were developmental 
and hierarchical within themselves, as intended. 15,16 There was some overlap between levels, which 
should be expected, especially for frameworks designed to provide measurements against a 
continuous developmental scale. 

What is the relationship between the ALLS and ACSF frameworks? 

When comparing the levels across frameworks, a similar pattern was seen for both the reading and 
numeracy constructs. That is, Level 1 on both frameworks appeared to be similar in their complexity, 
whereas Levels 2 and 3 on the ALLS were found to be more complex than ACSF exit Level 2 and exit 
Level 3 respectively in both constructs, with the difference more pronounced for reading. That is, 
ACSF reading exit Level 3 appeared to be more similar to ALLS reading Level 2 than ALLS Level 3, and 
ACSF reading exit Level 4 was closely aligned to ALLS reading Level 3. For numeracy, the difference 
between frameworks at Levels 2 and 3 was still evident, but not as pronounced as seen in the reading 
construct. The indicative empirical relationship between the levels is summarised in table 1. 


Table 1 Empirical alignment of ACSF to ALLS by ACSF level 


Reading 

ACSF level 

ALLS level 

Numeracy 

ACSF level 

ALLS level 

1 

1 

1 

1 

2 

1-2 

2 

1-2 

3 

2 

3 

2-3 

4 

3 

4 

3^i 

5 

4-5 

5 

Uncertain 


The small number of ratings made against Level 5 numeracy items meant that the relationship 
between the ALLS and ACSF frameworks at this level could not be determined reliably; hence, the 
inclusion of ‘uncertain’ in the table. 


14 See ‘Empirically validating the measures used for each framework’ in the Results section of the support document, 
Mapping adult literacy performance: support document. 

15 Due to the limited number of respondents rating items at Level 4 and, in particular, Level 5, extra caution is needed 
when making observations at these levels. 

16 Only seven of the 165 items had complexity estimates that were not consistent with their designated levels in the 
frameworks: four in the reading construct (three from the ALLS; one ACSF item), and three in the numeracy construct 
(one ALLS item; two from the ACSF). These items were subsequently excluded from further analysis where the 
complexity levels across frameworks were compared. See table A2 for further detail, or refer to ‘Empirically validating 
the measures used for each framework’ in the Results section of the support document, Mapping adult literacy 
performance: support document. 
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Discussion 


So, does 1 = 1? This study has shown there to be a close alignment in the complexity of Level 1 
reading and numeracy constructs between the Adult Literacy and Life Skills survey and the Australian 
Core Skills Framework. However, the alignment between each performance level across the two 
frameworks was not as direct for higher skill levels. For example, as we have seen, for the reading 
construct, ACSF exit Level 3 appeared to be more similar to ALLS Level 2 than ALLS Level 3, and ACSF 
exit Level 4 was more closely aligned to ALLS reading Level 3. A similar outcome was found for the 
numeracy domain, although not as pronounced. 

The impetus for this project came from two directions. The first was the National Skills and Workforce 
Development Agreement (Council of Australian Governments 2008), which specified that the 
proportion of the working-age population with low foundation skill levels be reduced to enable 
effective educational, labour market and social participation, and that the proportions at ALLS Levels 
1, 2 and 3 be monitored as a means of checking progress. The second was the aspirational target of 
the 2012-22 National Foundation Skills Strategy for Adults. 17 In this context, what do these findings 
mean? If, as the results are suggesting, ALLS Level 3 in reading and numeracy is approximately 
equivalent to ACSF exit Level 4, then adult literacy and numeracy programs that are delivered and 
reported against the ACSF may need to specify ACSF exit Level 4 as the desired outcome if the implied 
workforce skills development objective is to be met. This may have consequences for current 
programs and the extent and ways in which they are resourced. 

Providing adults with the opportunity to achieve a prescribed benchmark adult literacy and numeracy 
performance level, perhaps ACSF exit Level 4, may require a greater investment in literacy and 
numeracy support per individual than is currently being committed. It may also require additional 
hours of instruction and a larger adult literacy workforce than currently exists. This being said, with 
finite resources, priorities will need to be established to ensure that the individuals who are likely to 
benefit the most are targeted. 

Data capture 

With the broadening application of the Australian Core Skills Framework, its alignment to the Adult 
Literacy and Life Skills survey provides an opportunity for policy-makers to monitor the skill level of 
particular cohorts on a more regular basis and to examine which programs work better for different 
groups of learners and why. However, this does pose questions about how we might best capture ACSF 
data to use as evidence of literacy and numeracy progress against the Adult Literacy and Life Skills 
survey and its successors. As a starting point, existing data collection tools, such as the Language, 
Literacy and Numeracy Program Information System, may require modifications. 

The findings of the current study also have implications for the future use and refinement of the 
ACSF, as well as future mapping to other similar programs (for example, Adult Migrant Education 
Program) and frameworks (for example, the Core Skills for Work Framework). These are discussed in 
more detail below. 


17 The aspirational target for the National Foundation Skills Strategy for Adults is that by 2022, two-thirds of working- 
age Australians will have literacy and numeracy skills at Level 3 or above, where Level 3 refers to the ALLS/PIAAC 
skill levels. 
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Empirical mapping of the ACSF and ALLS at Levels 4 and 5 

With very few ratings of learners at Levels 4 and 5 in both reading and numeracy, there was less 
certainty about the true complexity estimates of statements drawn from the higher levels of both the 
Australian Core Skills Framework and the Adult Literacy and Life Skills survey. The low number of 
ratings may have been due to respondents being less familiar with the reading/numeracy skills of 
learners at the higher levels. This is not necessarily unexpected, as it is less likely that learners with 
higher ability (in terms of reading and/or numeracy) would undertake specific adult literacy and/or 
numeracy programs such as the Language, Literacy and Numeracy Program. 

In order to obtain more accurate measures at the higher levels on the Australian Core Skills 
Framework and to compare these with the Adult Literacy and Life Skills survey, a future study could 
be undertaken in which individuals judge their own level of ability directly against the statements. 
(Hence, the survey would be designed to resemble a self-assessment as opposed to a performance 
judgment of a learner.) A similar methodology was employed to empirically validate the 
Developmental Learning Framework for School Leaders 18 in Victoria, in which aspiring school 
principals were surveyed using a self-assessment tool. If the survey instructions for the current study 
were redesigned to reflect a self-assessment, the target population for calibrating the ACSF and ALLS 
at Levels 4 and 5 could include the respondents from this study (adult literacy and numeracy 
educators and specialists), as well as students undertaking undergraduate and postgraduate studies, 
majoring in English and/or mathematics. 

An assessment tool based on the foundation skills frameworks (the ACSF and the Core Skills for Work 
Framework) will be developed by the Australian Government in 2012 and may provide a basis, or have 
implications, for such work. 

Improvements to the structure of the ACSF 

This study has demonstrated that the Performance Features within the ACSF have varying levels of 
complexity, even within each level of the framework. The implications of such findings suggest the 
potential for further improvements to the current structure of the ACSF. For example, three 
statements drawn from the ACSF were found to have unexpected complexity estimates, which should 
be reviewed when future revisions are made to the framework. The study has also shown that it is 
possible to recognise sub-levels within each of the five levels. That is, the empirical positioning of 
Performance Features in the current study illustrates that the results can be used to assist in 
describing the typical developmental pathway of learners within an ACSF level. If a similar follow-up 
study was undertaken with a larger pool of Performance Features sampled, then it would be possible 
to describe what it means to be at a higher or lower level within each of the existing five levels of the 
ACSF. (For example, a profile description could be developed to describe a typical high-level ACSF 2 
learner as opposed to a typical lower-level ACSF 2 learner.) Hence, it is possible that the ACSF could 
be extended to have profile descriptions within each level (two sub-level profiles per level). 

The confirmation of the hierarchical structure within ACSF levels is in itself an important finding, as it 
offers the possibility of being able to clearly demonstrate progress within a level. This may be 
particularly important for both providers and learners, especially learners with lower skill levels, as 
demonstrable progress within a level may be a more realistic goal than progress between levels. 


18 See <http://www.eduweb.vic.gov.au/edulibrary/public/teachlearn/leader/developmental_learning. 
framework_20070418.pdf >. 
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Application to other frameworks 

This study has demonstrated that developmental learning frameworks that are considered to be 
measuring the same overarching constructs can be empirically positioned onto the same measurement 
scale using a combination of complex survey design methodology and Item Response Theory analyses. 
Moreover, the study has shown that such a methodology can also be used to empirically validate 
developmental frameworks (such as the ACSF) to ensure that they are hierarchical, developmental 
and cumulative, as intended. Prior to conducting the current study, it was assumed the ACSF had five 
levels of increasing complexity in reading and numeracy; this study now confirms this assumption. 

The results from this study could be used to map other similar frameworks or programs onto the 
Australian Core Skills Framework and/or the Adult Literacy and Life Skills survey. For example, the 
Adult Migrant English Program (AMEP), if considered to have similar constructs in terms of 
reading/numeracy, could also be mapped onto the Reading and/or Numeracy complexity scales 
developed in this particular study. Linder such circumstances, only a sample of the calibrated 
statements used in this study would need to be included in the follow-up study, as well as a sample 
of additional statements (for example, learning outcomes) drawn directly from the Adult Migrant 
English Program. 

Similarly, the new, yet to be released Core Skills for Work Framework (CSFW; ITHACA Group 2012), 
which has been designed to have five developmental levels across ten skill areas (to complement the 
ACSF), could also be empirically validated using a similar methodology to that employed in the 
current study. The empirical validation of this framework would enable the developmental, 
hierarchical and cumulative nature of the five levels within the framework to be tested per skill area, 
as well as minimise any redundancies and/or ambiguity in the Performance Features that may exist 
within and across the ten skills sets. In addition to empirically validating the framework in terms of its 
architectural structure etc., it may also be desirable to map certain skills sets within its framework to 
the ACSF. 

Conclusion 

This study has demonstrated the empirical alignment between the Australian Core Skills Framework 
and the Adult Literacy and Life Skills survey. But if, for a moment, we take a step away from the 
focus on the degree of equivalence between the two frameworks, there is a more fundamental issue 
that needs considering. Is the stipulation of skill levels, as measured by the Adult Literacy and Life 
Skills survey (or the Programme for the International Assessment of Adult Competencies), as a means 
of monitoring progress against the National Skills and Workforce Development Agreement and, indeed, 
the key performance indicator for the new National Foundation Skills Strategy for Adults, the most 
appropriate indicator of the work being done within Australia to help people develop their literacy 
and numeracy skills? 

Do we need a paradigm shift? Instead of nominating a fixed benchmark to aspire to, could measures of 
success be shown by progression in literacy and numeracy skills, for example, either within or 
between levels of the Australian Core Skills Framework? This may be a more suitable or appropriate 
position to take, since progressing between ACSF pre-Level 1, Level 1 and Level 2 (and therefore, the 
ALLS equivalent of Levels 1 and 2) may take relatively longer than would progressing between ACSF 
Levels 2 and 3 and higher, given the initial skill levels of the learners. 
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This is not to suggest that indicators such as the Adult Literacy and Life Skills survey or the 
Programme for the International Assessment of Adult Competencies are irrelevant. These indicators 
are important in the international context in providing a comparable measure of Australia’s position 
against other OECD countries. But the use of these international indicators should not preclude the 
use of indicators that hold greater contextual meaning for measuring the literacy and numeracy skills 
of Australian adults. A current example of where both national and international indicators are used 
to gauge progress in meeting educational outcomes occurs in the Australian school sector. The 
National Assessment Program includes the National Assessment Program - Literacy and Numeracy 
(NAPLAN); sample assessments in Science Literacy, Civics and Citizenship, and Information and 
Communication Technology (ICT) Literacy conducted on a three-year rotation; and participation in 
international sample assessments, namely, the Programme for International Student Assessment 
(PISA), the Trends in International Mathematics and Science Study, and the Progress in International 
Reading Literacy Study. 

With the Australian Core Skills Framework increasingly being used in a variety of settings beyond the 
Language, Literacy and Numeracy Program and the Workplace English Language and Literacy program, 
the alignment demonstrated by this research offers the potential for more detailed and regular 
monitoring of progress against national objectives. 
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Appendix 


Figure A1 State/territory of respondents 


ACT 

1% 



n=6 

NSW 


23% 


n=96 

NT 

5% 



n=22 

QLD 

6% 



n=24 

SA 

9% 



n=36 

TAS 

8% 



n=31 

VIC 



40% 

n=163 






WA 

8% 



n=33 


Figure A2 Description of respondents' workplaces 


Education &/or Training.. 


80% n=330 

Not for profit organisation 

- 

10% 

n=42 

Private enterprise 

3% 

n=13 

Private consultant 

2% 

n=9 

Public Service 

3% 

n=1 1 

Other 

1% 

n=6 


Table A1 Number of respondents by workplace subcategory 


Workplace 

Total number of 
respondents 

Workplace 

category 

Number of respondents 
for each category 

Education &/or training 

330 

TAFE 

185 

institution 


Community-based RTO 

21 



Dual/multi-sector 

28 



Private RTO 

32 



School sector 

9 



University 

55 

Not-for-profit organisation 

42 

With RTO status 

37 



Without RTO status 

5 

Private enterprise 

13 

With RTO status 

10 



Without RTO status 

3 

Public service 

11 

Federal 

1 



State 

10 
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Figure A3 Years of experience delivering adult language, literacy or numeracy training 


Less than 1 year 


7% 


1 -5 years 


27% 


n=30 


n=1 1 3 


6-10 years 


21 % 


n=87 


More than 10 years 


44% n=1 81 


Table A2 Complexity of estimate outliers 


Construct 

Framework 

Level 

Unexpected complexity estimate 

Reading 

ALLS 

3 

Easier than expected 

'Locates relevant information in a text where the information is 
literal or requires only low level inference.' 


ALLS 

4 

Easier than expected 

‘Write in your own words one difference between the panel 
and the group interview.’ 


ALLS 

5 

Easier than expected 

‘List two ways in which CIEM (an employee support initiative 
within a company) helps people who lose their jobs because of 
departmental reorganization.' 


ACSF 

4 

Easier than expected 

‘Actively identifies an explicit purpose for reading, e.g. to 
gather background information, identify specific facts or 
understand a concept.’ 

Numeracy 

ALLS 

2 

Easier than expected 

‘Identifies and understands basic mathematical concepts in a 
range of familiar contexts where the mathematics content is 
quite explicit and visual with little distracting information.’ 


ACSF 

1 

Flarder than expected 

‘Recognises and compares familiar basic metric 
measurements and quantities such as length, mass, 
capacity/volume, time, temperature, e.g. personal height and 
weight, a litre of milk or vehicle height clearances.’ 


ACSF 

1 

Flarder than expected 

‘Compares information and data within highly familiar simple 
texts, lists, charts, diagrams and tables.’ 
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Support document details 

Additional information relating to this research is available in Mapping adult literacy performance: 
support document. It can be accessed from NCVER’s website 
<http://www.ncver.edu.au/publications/2581 .html>. Details include: 

■ Overview 

■ Introduction 

■ Methodology 

■ Survey design 

■ Results 

■ Conclusion. 
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